chore: sync upstream main through f8556dd0 - #68

Nodes now compute a deterministic capability hash over their canonicalized
hardware report (resolved backend, render devices with PCI address and NVIDIA
uuid, host boot_id, probed backends, transformations, tone-map executors) and
advertise it in the 30s health response without ever running probes there. On
a hash change the health sweep fetches /hw-capabilities detached from the
sweep (deduplicated per node, bounded at 2m to cover a cold node's probe
budget), persists the payload to new nullable stream_nodes columns, updates
the in-memory pools copy-on-write, refreshes the v3 capability cache, and
logs capability drift. Nodes without a hash keep today's behavior exactly.

GET /admin/nodes now returns the stored inventory plus derived
physical_gpu_keys (gpu_uuid, else boot_id|pci_address) for shared-GPU
detection, and the admin Nodes page gains a GPU column with verified/failed
backend badges, device summaries, and a staleness indicator tied to the
health-check clock. /admin/nodes and /admin/system/hw-accel are now
documented in docs/admin-api.md.

Phase 2 of the node GPU observability plan.

Related issue: Silo-Server#780

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Registry-driven 12-column widget grid with customize mode (drag reorder,
drag-to-resize with keyboard alternatives, add/remove sheet), layout
persisted to localStorage. Old dashboard sections extracted into widgets;
Trakt card replaced by a compact sync strip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

New internal/nodemetrics sampler on nodes and the API host: CPU (cgroup
quota-corrected) with load and cores, memory via the meminfo/cgroup readers
extracted from postgres_tune, network throughput, and Statfs disk usage on
the transcode scratch dir and library roots — every mount probed in its own
bounded goroutine so a hung network filesystem can never block sampling,
health, or a scrape. GPU usage comes from DRM fdinfo of the node's own
ffmpeg children, tracked per DRM client and summed per device, with
nvidia-smi enrichment behind a circuit breaker; per-device session counts
now include single-device QSV/VAAPI and NVENC workloads and join the
sampler through a device alias set.

Health responses carry the sample (bounded at 256KiB body / 32KiB stats);
the 30s health write persists it to a new stream_nodes.last_stats column;
node modes mount /metrics with disk series labeled opaquely so library
paths stay off the unauthenticated surface; GET /admin/system/resources
serves the API host's own sample. Admin UI gains the Nodes system column,
live GPU busy/session numbers, and a dashboard server-resources card.

Also: hardware detection now skips configured hw_device entries this
process cannot open — classified for reporting (detected_backends gains
"skipped") but never smoke-encoded — so proxy nodes reading the
cluster-wide hw_device stop probe-failing with driver errors and the Nodes
page no longer warns about them. Branch-new lint findings cleaned across
the touched packages.

Phase 3 of the node GPU observability plan.

Related issue: Silo-Server#780

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

New nullable stream_nodes columns hw_accel_override (enum-checked) and
hw_device_override let a heterogeneous pool configure acceleration per node;
NULL inherits the cluster-wide playback settings. A node overlays its own
row's overrides onto the cluster config on every nodeconfig reload, matched
by NODE_URL against the unique url column (trailing-slash tolerant,
deterministic on ties, conservative on lookup failure). Remote transcode
dispatch sends Node.EffectiveHWAccel — the override when set, else the
cluster value — so auto still reaches the node for live resolution and a
stale capability report can never pin a backend; jellycompat dispatch gains
the same rule via a planner node lookup.

PUT /admin/nodes/{id} accepts both fields with explicit-null clearing and
case-insensitive enum validation (400 on bad values). The node edit dialog
gains "Inherit cluster setting" controls, the GPU cell shows the override
source, and the Playback Settings divergence warning now points at per-node
overrides. Changes hot-apply in stages (dispatch immediately, node config
within a reload, snapshots within 15m); only boot-time warmup and in-flight
sessions wait for a restart — documented in docs/admin-api.md.

Phase 4 of the node GPU observability plan.

Related issue: Silo-Server#780

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Physical GPU identity moves into nodepool: every stored or pooled node row
derives physical_gpu_keys from its capability payload (NVIDIA uuid, else
boot_id|pci_address; a device with neither — including a missing boot id —
contributes no key rather than a cross-host-colliding one). Transcode
selection gains a strictly secondary tie-break: when candidates are level on
effective jobs, the node whose physical GPU group (itself plus every pooled
node sharing a key) carries fewer total jobs wins. Primary least-jobs
ordering, session soft-affinity, and proxy selection are unchanged; job
counts only, no utilization input.

The admin Nodes page shows a Shared GPU badge naming the nodes a card is
shared with, across the transcode and proxy tables. docs/admin-api.md
documents the derivation, its boot-scoped stability caveat, and the planner
behavior.

Phase 5 of the node GPU observability plan.

Related issue: Silo-Server#780

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

On split-horizon topologies the registered stream_nodes.url is the public
address the API dials while NODE_URL is the node's internal one, so the URL
match for the per-node override row can never hit. When the URL matches no
row, the watcher now falls back to matching NODE_NAME against the registered
name; an ambiguous name (no unique constraint) matches nothing and warns
once. URL matches keep precedence. docs/admin-api.md documents the identity
contract: keep registered names unique and NODE_NAME equal to them.

Found validating phase 4 on the shared dev deployment, where both identities
diverged and overrides silently inherited.

Related issue: Silo-Server#780

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A node running in Docker nested inside an LXC sees the kernel's raw /proc
(bare-metal totals) and an unlimited own cgroup, because the LXC's cap lives
on an ancestor cgroup invisible from the nested namespace — so cpu, cores,
load, and memory reported the host. The sampler now prefers
/host/proc/{stat,loadavg,meminfo} when present, which deployments bind-mount
from the LXC where lxcfs virtualizes them to the container's real limits.
net/dev and per-PID fdinfo reads deliberately stay on the container's own
/proc. Documented with a compose snippet in the docker guide.

Related issue: Silo-Server#780

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ng, device picker

Re-probe: exported probe-cache invalidation in playback and tonemap, a
bearer-authed POST /admin/reprobe-capabilities on transcode and proxy nodes
(refuses with 409 while transcodes run so a loaded smoke encode can never
publish a false hardware regression; 503 keeps the previous hash), and
POST /api/v1/admin/nodes/{id}/reprobe which extends the connection write
deadline past the probe budget and immediately refetches and persists the
new report through the health checker's existing machinery.

Scratch admission: the sampler marks the scratch disk in the payload and the
planner soft-excludes transcode candidates at >=95% scratch fill — never
emptying the candidate set (degraded beats down), conservative on missing or
stale stats, with latched transition logging that says whether the guard
actually excluded or was dropped.

Drift: the capability drift the sweep already computes persists to a new
nullable stream_nodes.capability_drift column (UTF-8-safe 512-byte note,
cleared only when every attempted probe passes again) and surfaces as a
warning badge on the Nodes page.

UI: hw_device_override becomes a checkbox picker fed by the node's stored
device inventory (shared parser extracted to lib/hwDevices, free-text
fallback without inventory, unknown configured paths preserved) plus a
per-row Re-probe action. New docs/wiki/admin/monitoring-nodes.md operator
guide and admin-api.md coverage for the new endpoints and fields.

Related issue: Silo-Server#780

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…t default

Proxy nodes only remux and strip bitstreams, so the edit dialog no longer
offers Hardware Acceleration or GPU Devices for them (and a proxy edit no
longer sends override fields at all, so values set via the API are not
silently cleared). The inherit option is relabeled "Cluster default" with
copy stating plainly that the default is auto and auto detects this node's
own hardware, plus a muted line showing what the node currently resolves to
from its stored capability report.

Related issue: Silo-Server#780

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Follow-ups from the review of this branch. Each is a case where a surface
this PR added could publish or persist something that is not true.

Incomplete detection is no longer publishable. walkHWAccelBackends bounds
itself at 30s regardless of how many candidate devices a host exposes, and a
walk that ran out marked the backends it never reached Verified=false — which
is byte-identical to a real hardware failure. The transcode node's outer
budget (~106s) stayed healthy through that, so the degraded report was hashed,
advertised, refetched, and persisted as a capability_drift regression for
hardware that was fine, with the node routed to software until a clean probe
landed. Detection now reports completeness (ErrHardwareDetectionIncomplete)
and both node types refuse to publish a cut-short walk, keeping the previous
hash exactly as they already do for a failed tone-map probe. The proxy's
ctx.Err() check only caught the outer context, so it gains the same guard plus
one hardware-aware deadline over both of its probes, matching the transcode
node.

The verified render device now reaches execution. Detection walks a backend's
candidates and stops at the first that passes a smoke encode, but resolution
discarded that device: a transcode with no configured playback.hw_device fell
back to PickRenderDevice, which returns whatever sorts first under /dev/dri.
On a mixed-vendor host those are different GPUs, so "qsv verified" was paired
with ffmpeg initializing a card the probe never touched. The passing device is
recorded per probe generation and adopted by the allocator, which also closes
the reporting gap it caused: a default-configured node counted no workload for
an unnamed device and reported zero sessions beside a busy engine.

Probe-cache invalidation now supersedes in-flight probes. Both caches only
cleared their map, so a probe that started first completed, stored its verdict,
and handed it to a caller that had since invalidated — the operator re-probe
could republish exactly what it was asked to discard and report "nothing
changed". An invalidation generation in the cache and singleflight keys moves
the key instead; shared work is still never canceled.

The re-probe's busy check is now an exclusion, not a sample. activeJobs only
moves once ffmpeg is running, so a node idle at the check accepted a transcode
milliseconds later and the smoke encode raced the live encoder anyway — the
false regression the 409 exists to prevent. A gpuGate held for the whole
rebuild is consulted by every path that spawns ffmpeg (start, reconstruct,
prepared download); neither side ever waits, both refuse.

Drift recovery needs evidence. hardwareProbesClean returned true for a report
with no detected_backends at all, which is what a GPU that disappeared entirely
produces, so the next unrelated hash change (a reboot moving boot_id) cleared a
standing note and told the operator the node recovered. Recovery now requires
at least one probed backend that passed.

Also: library-N Prometheus labels are assigned before the unavailable-mount
skip, so a mount going away no longer renumbers the ones after it under an
unchanged label; the proxy's /status guards a nil tracker the way /health
already did; and docs/admin-api.md no longer says nothing in node selection
reads last_stats while documenting the scratch admission guard that does.

Found by review of Silo-Server#794, including findings raised by Codex and CodeRabbit.

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ide layouts

Builds the insights phase of the admin dashboard rebuild:

- internal/dashmetrics: minute-resolution sampler for concurrent streams
  (by play method) and egress (node egress + per-process viewer egress from
  stream telemetry), replica-safe via idempotent inserts, 31-day retention.
- New aggregate endpoints following the AdminStatsProvider pattern:
  /admin/stats/timeseries (adaptive peak-preserving bucketing, 1h-31d),
  /admin/stats/playback-activity (hourly/daily buckets from playback
  history unioned with live sessions, reliability counters),
  /admin/stats/top-activity (top titles and profiles from watch history).
- /admin/server/status gains a health object (Postgres/Redis pings,
  24h error and warning counts); /admin/logs/app accepts a comma-separated
  level list.
- Server-side per-admin dashboard layouts (admin_dashboard_layouts table,
  GET/PUT/DELETE /admin/dashboard/layout) with localStorage as the
  instant-paint fallback and one-time migration of existing local layouts.
- 13 new widgets (playback activity, streams/egress trends, reliability,
  top titles/profiles, nodes, scanner, scan activity, recent errors,
  health strip, and three stat tiles) on hand-rolled SVG chart primitives
  using re-stepped, accessibility-validated chart tokens.
- Widgets resize on both axes via a corner drag handle (12-column x
  100px-row grid, keyboard accessible), and metric widgets carry a
  persisted 1h/24h/7d/30d range picker.
- Users widget sorts by last activity instead of account id.

Time-to-first-frame and failed-start metrics are documented as future
work; no data source exists for them yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Three follow-ups from the review of cca2f45.

Filesystem paths leave the unauthenticated surface. A node's /api/v1/health
takes no credential, and the resource sample this branch added to it carried
`system.disks[].path` — the transcode scratch volume, and on the API host every
configured library root. That is deployment layout rather than a host resource
counter, and it is exactly what /metrics already withholds by labeling disk
series with a role. DiskStats gains that Role ("scratch", "library-N"),
assigned once when the sample is built, and health serves a path-free
projection (Snapshot.RedactPaths). Paths stay on the surfaces that require a
credential: each node's bearer-authed /status and the admin-authenticated
/admin/system/resources. GPU device names are deliberately kept — a render node
or a CUDA index is a fact about the hardware, not about this deployment, and
/metrics already labels its per-GPU series with the same value.

Assigning the role at sample time also makes the positional-label guarantee
structural rather than a call-ordering subtlety in the collector: the index
belongs to the mount, so one going unavailable no longer renumbers the ones
after it, and health, /status and /metrics all name a mount identically.

Disk probing is bounded, not just disk reporting. Every configured mount got
its own statfs goroutine every five seconds while only eight entries could ever
be published, and statfs on a dead network mount is uninterruptible — the
goroutine parks until the mount recovers or the process exits. A deployment
with forty library roots therefore started forty probes per interval to fill
eight slots and could park one goroutine per unreachable root indefinitely. The
cap now applies to what is probed. The scratch dir is always first and so is
never the entry dropped, and roots past the cap are logged rather than silently
omitted.

Also: the playback-to-nodemetrics device-identity adapter had three identical
copies (both node servers and cmd/silo). It is now playback.SamplerDeviceIdentities,
which is where the conversion belongs — nodemetrics stays free of any playback
import, which is why it takes the identities as a provider at all.

Found by review of Silo-Server#794, including findings raised by Codex and CodeRabbit.

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two follow-ups from the review of f4b099d.

The in-flight invalidation test ordered its race with a 50ms sleep, which
CLAUDE.md rules out ("wait on observable state — job status, health endpoints,
channel receipts — not fixed sleeps") and which would fail on a loaded machine
for no production reason: if the flight had not registered yet, both calls
would share a generation and the second would join the first. The probe now
has the same start seam the tone-map probe cache already offers its test
(hwProbeFlightStarted, nil in production): the first flight parks inside the
probe until the invalidation has landed, so the test is decided by channel
receipts. The fake ffmpeg's artificial delay and the raised command budget it
needed are both gone.

A prepared download claimed the GPU gate before checking whether the artifact
already existed, so a re-probe refused requests that would have been served
from disk without touching the encoder. The claim moves below the reuse check
and above the tone-map recipe resolution, which is the first thing in that
handler that actually runs ffmpeg.

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The Go check has been red on this branch since 189d050: seven proxy capability
tests fail on CI and pass on a developer's machine. They configure no
playback.ffmpeg_path, so the probes shell out to whatever `ffmpeg` is on PATH,
and the CI image installs none — ProbeTransformationRegistryWithToneMapV3Result
reports the exec failure, buildCapabilitySnapshot returns it, and the endpoint
answers 503 where the test asserts 200. It reproduces locally with an empty
PATH.

They now point at a scripted binary with a known successful answer for each
listing the assembly runs, which is how the transcode node's capability tests
already avoid the same dependency. That also makes the published hash
deterministic, which is what the stability assertions were already relying on
the host to provide. newDownloadProxyServer is unchanged for the download tests
that never exec ffmpeg.

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Four follow-ups from the review of cfd444c.

The re-probe exclusion had two holes left. Hardware chapter-thumbnail
extraction reserves a render device and runs ffmpeg on it but never touches
activeJobs, so it left the node looking idle to a re-probe; it now takes the
same gate the transcode, reconstruct and prepared-download paths do. Separately,
the gate only ever covered work — not other capability builders — and the probe
caches deliberately no longer coalesce a re-probe with a build already in
flight, since bumping the invalidation generation is what makes the re-probe
honest. Those two together meant the scheduled 15-minute snapshot (or an
authenticated /hw-capabilities request) could run its ffmpeg smoke matrix beside
the operator's on the same GPU, which on session-limited hardware is exactly the
collision that publishes a false regression. Capability assemblies now serialize
on one lock, which the re-probe holds across its invalidation and its rebuild so
the two are a single step.

The disk-probe cap bounded paths per sample but not probes overall. A wedged
probe is deliberately kept — dropping its entry would only let the next sample
park a second goroutine on the same mount — so a deployment whose library roots
churn while mounts are dead would retire one set of parked goroutines' paths and
immediately be free to park a fresh set. There is now a ceiling on probes
outstanding at once across every path ever offered, logged when it bites, which
makes that a fixed cost rather than a growing one. Paths are offered scratch
first, so admission control's mount gets any freed slot first.

Finally, an override update now asks the node to re-read its configuration
before the pools are reloaded. Dispatch names the new backend as soon as the
pool reloads while the node re-reads its own row on a 60-second poll, so
changing both overlays at once — QSV on a render node to NVENC on a CUDA index —
paired the new backend with the old device for up to a minute. The nudge is best
effort and only fires when an acceleration field actually changed; the node's
own poll remains the backstop.

Found by review of Silo-Server#794 (Codex).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two follow-ups from the review of 0765cea.

A capability fetch runs detached from the sweep and is bounded at two minutes,
which is ample time for an administrator to repoint a node's URL at a different
machine. The write keyed on the id alone, so one worker's GPU identities could
land on a row that now addresses another — and the planner's shared-GPU
tie-break and the admin inventory would use that reading until a later sweep
corrected it. UpdateCapabilities now also matches the URL the payload was
fetched from and reports ErrNodeMoved when it does not, which the sweep logs and
discards rather than publishing. Trailing slashes are ignored on both sides
because the pools normalize URLs and the column does not.

Drift compared render devices by enumeration path, and DRM is free to hand the
same card a different renderD number across a reboot. That reported the card as
gone — and because the reboot also moves boot_id it triggered the refetch that
persists the note, so an operator saw a hardware regression for a GPU that never
moved, latched until the next clean report. Devices are now matched on the
strongest identity each one publishes: the NVIDIA uuid, else the PCI slot, else
the path for a node that reports no details. A card that genuinely disappears
still has neither, so it is still caught.

Found by review of Silo-Server#794 (Codex).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…emory reads

Three follow-ups from the review of 745f0d8 and ac3cc23. The first is a
regression the override nudge in 745f0d8 introduced.

/admin/force-reload on a transcode node closes every live playback session and
removes its output directory and recipe. That is the right answer when an
operator asks for it, and the wrong one for the control plane's own
housekeeping — and the admin form posts both override fields on every
transcode-node save, so the nudge fired on any edit at all. Renaming a node or
changing its capacity interrupted every stream on it, while the UI said the
policy applied to new transcodes. Nodes now expose a non-destructive
/admin/reload-config that re-reads configuration and nothing else, which is what
the nudge calls; the API also compares the stored row before and after the write
and stays silent unless a value actually moved. Watcher.ForceReload now returns
an error instead of dereferencing a nil pool, since an operator-reachable route
should not panic on a watcher built without a database.

Drift matched GPUs on the single strongest identity each device published, but
identity strength is not constant: nvidia-smi sits behind a circuit breaker, so
the same NVIDIA card alternates between reporting a PCI address alone and
reporting a uuid as well. Those two reports compared as different devices and
persisted a "render device gone" note for a card that never moved — the same
false positive as comparing enumeration paths, one level up. Devices now carry
every alias they publish and match when they share any of them.

Memory paired a cgroup working set with host RAM. A container with no memory
limit still publishes a readable memory.current, so a node reported something
like 1 GiB used of 64 GiB while the host was nearly out of memory. The cgroup
usage is now read only when the total beside it is the cgroup's limit, so both
numbers always describe one domain.

Found by review of Silo-Server#794 (Codex).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…nd fences

Five follow-ups from the review of 41222fa, plus a route manifest that commit
left stale.

NVENC was always smoke-tested with an empty device while execution passes the
configured hw_device through as -hwaccel_device. A working GPU 0 could therefore
verify NVENC on behalf of a configured GPU 1 that is absent or broken, and the
real transcode then failed — the same mismatch the render-device fix closed for
QSV and VAAPI, still open on the NVENC branch. The probe now uses the CUDA
identity execution will use. Openability is not consulted for it: a CUDA index
or GPU uuid is not a file, so failing to open it says nothing, and the smoke
encode is the only thing that can answer.

The per-node device picker offered render paths whatever the backend, so
switching a node to NVENC left it holding a /dev/dri path with no way to enter
the CUDA identity NVENC needs. It now follows the same rule the cluster-wide
Playback form already applies, falling back to a free-text field with
NVENC-appropriate help.

Drift matching let weaker aliases override conflicting uuids, so a replacement
card in the same slot — same PCI address, usually the same render path — hid the
old card's disappearance entirely. Two permanent uuids that disagree are now
decisive; the weaker aliases only stand in when at least one side published no
uuid, which is what keeps an appearing or disappearing uuid from reading as
drift.

Health writes are now fenced on the checked URL, as capability writes already
were. The window is smaller — a health request is bounded at five seconds — but
last_stats carries the scratch fill transcode admission reads, so a sample
landing on a repointed row can exclude a healthy node or admit a full one.

Ordinary capability snapshots now register as GPU work, so a manual re-probe
cannot claim an apparently idle encoder and run its matrix beside one. They
deliberately still do not refuse while transcodes run: a node under sustained
load would then never refresh its inventory, which is worse than the cold-start
contention it would avoid.

Found by review of Silo-Server#794 (Codex).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The Web check's format step failed on AdminNodes.tsx: the NVENC placeholder
ternary added in fc99adc runs past the 100-column printWidth. Prettier's own
output, no behavior change.

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…eams

Five follow-ups from the review of fc99adc, all in code this branch added.

The config-reload nudge accepted only 200 while both node handlers answer 204,
so every *successful* reload logged a warning saying the node had refused and
would wait for its polling fallback. That is a standing false alarm on the
ordinary path — exactly the kind of signal an operator learns to ignore. Any 2xx
now counts as success.

recordVerifiedHWDevice read the current generation when it wrote, while the
probe that earned the verdict captured its own before starting. An invalidation
landing mid-probe would therefore file the device under the new generation, and
VerifiedHWDevice would hand execution a device the re-probe was asked to
re-verify. The captured generation is passed in and a stale one is dropped,
which reads as "nothing verified yet" — the state a cold process is in, and the
one the next walk repairs.

The two tests added in 745f0d8 ordered themselves with a 200ms negative timeout
and a polling sleep, which CLAUDE.md rules out and which can only ever say "it
had not finished yet" — also true when it never started. The builder now reports
when it has claimed the work slot, and records on the far side of the lock
whether the test had already released it, so both are decided by receipts.
(Removing them also cleared a block this rewrite had duplicated.)

The per-node device help split one sentence across a conditional, so JSX could
render "Leaveempty", and its shared tail claimed the node had reported no
inventory — untrue for exactly the NVENC case that reaches it, since NVIDIA
hosts do publish /dev/dri entries. Each branch is now a whole sentence.

The disk-probe docs said the process logs which roots go unsampled; it logs how
many.

Found by review of Silo-Server#794 (Codex and CodeRabbit).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… URL

The database fence added in fc99adc and 41222fa guards the wrong side on its
own. checkAll publishes to the in-memory pool *before* UpdateHealth evaluates
the URL, so an administrator repointing a row and reloading the pools during the
five-second health request gets the old worker's health — and the scratch fill
transcode admission reads — written onto its replacement by id. UpdateHealth
then correctly refuses, but nothing undoes the pool write, and the routing pool
stays wrong until a later sweep: a full replacement node can be admitted, or a
healthy one excluded.

Both pool writers now take the address the result was obtained from and ignore a
result whose id matches but whose URL does not. Comparison goes through the same
normalization the pools apply, so a trailing slash is not a different worker.
The capability writer gets the same treatment: its fetch is bounded at two
minutes rather than five seconds, so its window is wider still, and what it
publishes is the GPU identity the planner places shared work on.

Found by review of Silo-Server#794 (Codex).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ng the inherited device

Two follow-ups from the review of 4423d7e.

An acceleration override change reloaded the worker and the pool but left this
server's protocol-v3 planning cache valid for its own TTL. That cache holds the
tone-map executors and transformation inventory the *previous* backend
advertised, so a session started in the next minute was planned against filters
the worker had already moved off — and the worker then rejected the start. The
update now drops that node's entry through the same invalidation the health
sweep uses, before the pool reload exposes the new policy, and only when an
override actually moved.

The device field also claimed to know what leaving it empty resolves to. Empty
inherits the cluster-wide playback.hw_device verbatim, and this form does not
know that value: "CUDA device 0" was wrong whenever the cluster names a render
path — which is the case that matters, because that path then reaches NVENC as
a CUDA identity and fails. The picker's "auto-discover this node's devices"
overclaimed the same way for an explicitly configured cluster. Neither names a
default now; the NVENC branch says plainly that an inherited value has to be a
CUDA identity for NVENC to use it.

Found by review of Silo-Server#794 (Codex).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The sampler treats the media-root set it is given as the whole truth:
refreshDisks prunes every path outside it — dropping the cached capacity
readings with it — and diskStats omits them from the sample. Returning nothing
on a failed DistinctLibraryPaths therefore did not "keep the previous pass's
mounts reporting" as its comment claimed. A two-second database hiccup blanked
every library mount from the admin resource panel and from Prometheus, and left
the next pass reporting them unavailable until fresh probes landed, all while
the mounts themselves were healthy.

The provider now reuses the last set the database actually answered with. An
empty result it genuinely returned is cached like any other, since an operator
who removed their last library has no roots. The caching wrapper is split out so
it is testable without a database.

Swept the sampler's other providers for the same shape while here: none need it.
FFmpegChildren reads this process's own /proc, DeviceIdentities globs /dev/dri,
and DeviceSessions is an in-memory snapshot — their failures are permanent
rather than transient, so there is no last-good answer to hold. Recorded that on
Options.MediaRoots so the asymmetry reads as deliberate.

Found by review of Silo-Server#794 (Codex).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

reloadNodeConfig swallowed its outcome, so an override edit whose nudge timed
out or was refused looked identical to one the node had adopted. It now returns
whether the node confirmed, and the caller names the consequence: until that
node's own poll catches up, its backend comes from this server's pool while its
device still comes from its own configuration, so a start dispatched to it in
that window can pair the two wrongly and fail.

The policy is still published when the node does not confirm. Withholding it
would leave an override the operator has saved, and can see stored on the row,
never reaching dispatch at all — nothing else re-reads that column — which is a
silent permanent misconfiguration rather than a loud one bounded by the poll
interval and self-healing. That trade is recorded at the call site.

This narrows the window rather than closing it. Closing it means sending the
effective device alongside the backend so both come from one source instead of
two mechanisms with different timing; that is a change to the node start
contract and to the recipe-card rebuild path, which deliberately re-reads
HWAccel/HWDevice from node config today. Left for a maintainer decision rather
than folded into a policy edit.

Found by review of Silo-Server#794 (CodeRabbit).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

computeCapabilityDrift flattened Skipped into "not verified", so a backend that
had passed its probe and is now skipped was recorded as lost. Skipped means no
probe ran, because the node cannot open that backend's configured devices — a
statement about access rather than about hardware, and one the GPU column
already reports on its own.

It also contradicted hardwareProbesClean, which counts a skipped backend as
clean: the note was set by one rule and cleared by the other on the next hash
change, so it flapped with nothing having changed. The two now agree.

The distinction is "could not try" against "tried and the driver said no". A
backend that was probed and failed is still a loss, and so is one that stopped
being reported at all, which is what a card disappearing looks like — both are
pinned down by tests beside the new one.

Found by review of Silo-Server#794 (CodeRabbit).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

len() on a nil slice is zero, so the guard added beside it in 8463622 said
nothing. Caught by the changed-lines lint gate, which I had not been able to run
locally; golangci-lint is installed now and the same invocation CI uses reports
no issues across this branch.

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…o-detected device

Two follow-ups from the review of 940abea.

Clearing a drift note asked only that every backend the node probed passed. On a
multi-GPU node that lost one card the survivor passes just as cleanly, and once
the degraded report is stored the delta finds nothing lost ever again — so the
next unrelated hash change erased a standing regression while the card was still
missing. Clearing now also requires the report to have regained something the
stored one lacked: a backend that verifies again, or a device identity that is
back. The empty-inventory case fixed earlier was the same defect seen from one
side; this is the general rule.

Separately, an explicitly configured backend short-circuits resolution, so a host
running hw_accel=qsv with no hw_device never walks its hardware and had no
verified device to reserve — reporting zero GPU sessions for every transcode it
ran, the same hole the auto path had, on the branch that never probes. It now
falls back to the render node execution is about to pick anyway.

That fallback made two existing tests host-dependent: both asserted an empty
setting stays unresolved, which was only true because the machine running them
has no /dev/dri. They now point at an empty device directory so they assert the
invariant rather than the host. Swept the other AcquireHWDevice call sites in
tests: the rest are either already hermetic or NVENC, which returns before this
branch.

Found by review of Silo-Server#794 (Codex).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Third correction to this same rule, and the previous two were wrong in the same
way: they tried to infer recovery from a delta. They cannot. Once a degraded
report is stored, every later comparison is degraded-to-degraded and finds
nothing, so each fix reached for a different proxy for "better than before" —
first any passing probe, then any growth in the inventory — and each proxy had a
counterexample. A surviving sibling probes perfectly cleanly with its partner
still missing; an unrelated GPU added later grows the inventory without
repairing anything.

The note now records what it is standing for. A new nullable
stream_nodes.capability_drift_baseline holds the backends that must verify again
and the alias sets of the devices that must reappear, written in the same
statement as the note so it always describes it. Clearing checks the current
report against that, not against the previous one. Devices are kept as every
identity they answered to, so a card returning renumbered — or on a pass where
nvidia-smi did not answer — still matches. Successive losses accumulate: two
cards going one at a time must both return.

A note carried over from before the column existed has nothing recorded to wait
for, and a clean report clears it, so an upgrade does not strand one.

Found by review of Silo-Server#794 (Codex).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The test named for cache isolation built a second provider whose cache was never
populated, so it only asserted that an empty cache returns nothing — it could not
have failed if the clone were removed. It now mutates what a successful read
returned, forces the query to fail, and checks the fallback still yields the
original set; then mutates that fallback and checks again, covering both slices
a caller can get its hands on.

Also: the admin API table claimed capability_drift_baseline is present whenever
capability_drift is, which the same document contradicts three paragraphs later
for notes predating the column.

Found by review of Silo-Server#794 (CodeRabbit).

Related issue: Silo-Server#780

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The bulk resets under test sweep their whole table, and the shared test
database may hold rows from other tests or a populated snapshot. Each
test now snapshots every pre-existing row its reset would touch and
restores it on cleanup, so only the seeded fixtures change durably.

Verified by seeding decoy cached rows before the run and checking their
poster path, last_refreshed, chapter thumbnail path, and retry
timestamp all survive the tests byte-for-byte.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Sheet entries are draggable with a dedicated payload type; the grid's
reorder drop machinery accepts it, shows the same insertion indicator,
and places the widget with one layout update via addWidget(id,
beforeId?). The non-modal sheet stays mounted but fades
pointer-transparent during the drag — unmounting the drag source
cancels HTML5 drags in some engines — and drops are announced on the
existing aria-live channel. The + button remains the keyboard path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- The playback sessions union excludes live rows whose session already
  reached history: finalize writes history before deleting the sync
  row, and that interval (or a failed deletion) double-counted the
  session in buckets and reliability.
- Process egress samples are bucketed on the database clock like the
  shared row, merging same-minute ticks by GREATEST, so host clock skew
  cannot land streams and their egress in adjacent minutes or write a
  future row the server-anchored chart grid drops.
- The egress chart draws the total with the download subset as a second
  line instead of deriving a playback series: past the two-hour
  resolution the two per-bucket maxima are preserved independently and
  their difference is not any minute's playback rate. Documented.
- The watch-history activity filter is an allowlist of on-server
  sources (legacy, manual, playback, jellycompat): plugin watch
  providers store arbitrary keys in `source`, so the old denylist
  counted any new provider's imported backlog as local plays.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review follow-ups on the concurrency raise (Silo-Server#817):

Lease enforcement: only the download had a deadline; a hung encode or
upload could hold a job indefinitely, so nothing enforced the claim-page
lease math and an unstarted page tail could outlive its 15-minute lease
and be reclaimed and duplicated by another node. Every job now runs
under ImageCacheJobTimeout (2 minutes) end to end, the claim page drops
from 10 to 5 jobs per worker so a page's worst-case drain is 10 minutes
against the 15-minute lease, and the arithmetic is asserted in
TestImageCacheWorkerCount against the now-exported
ImageCacheLeaseDuration.

Memory bound: worker count derived from CPUs alone could put 48
concurrent jobs — each able to hold a 25 MiB download plus a full Go
decode of the original for thumbhash — inside a container with a small
memory limit. The pool is now also capped at one worker per 512 MiB of
the tightest detectable memory bound (GOMEMLIMIT, cgroup limit, then
/proc/meminfo), with the original pool of 2 as the floor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

bulkUpdateInBatches returns the rows already committed alongside an
error, but bulkResetSurface returned before adding them to stats, so an
interrupted bulk reset serialized zero requeued/cleared rows despite
having durably modified thousands. Counts are now recorded before the
error check in both phases, and a regression test interrupts the clear
phase to prove the requeue phase's committed rows stay counted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Native HTML5 drags do not scroll the page, so moving a widget to an
off-screen part of the grid was impossible. Both drag flows (reorder
and add-from-sheet) and corner resizes now edge-scroll: within 80px of
the visible top or bottom the scroll container moves, speed ramping
with proximity, driven by one rAF loop that stops on every
drag-termination path. The container is resolved by walking up from
the grid (document scroller today; an inner overflow region would work
unchanged). Resize sessions fold the scrolled distance into the
vertical delta, since pointer events do not re-fire while content
moves under a stationary pointer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Two more review follow-ups on the image-cache sizing (Silo-Server#817):

Cgroup limits: worker sizing consulted only the root-level cgroup
memory files, which are right inside a namespaced container but wrong
for a systemd unit with MemoryMax= or a leaf inheriting a tighter
slice/pod limit — those fell through to host MemTotal and could size 48
workers inside a small cgroup. nodemetrics already resolves this
process's own cgroup and walks its ancestors for the sampler; that
machinery is now exposed as nodemetrics.EffectiveMemoryLimitBytes and
used for sizing, with a fixture test covering the systemd-slice shape.

Overshoot: the per-job context timeout cannot preempt the synchronous
decode/encode segment (imageutil.Thumbhash and GenerateVariants take no
context), so the two-minute bound is not perfectly hard — the job stops
at the next context-aware step. That segment works on inputs capped at
25 MiB, so its overshoot is CPU-bounded; the claim page drops from 5 to
4 jobs per worker, keeping the worst chain inside the lease with nearly
two minutes of overshoot allowance per job, and the sizing test now
requires that headroom instead of a bare drain < lease check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Each batch restarted its ordered scan at the smallest key, so the
database rechecked every previously reset row — still in the key index
but no longer matching — before reaching the next batch, making the
sweep O(N²/batchSize); on 1.32M chapter-thumbnail rows that is
hundreds of millions of repeated predicate checks including
jsonb_array_elements evaluation. Both loops now carry the batch's last
key into the next batch's WHERE, so each key range is scanned once,
termination no longer depends on predicate falsification alone, and a
row re-cached by a concurrent writer behind the cursor is left for the
next reconcile instead of being reset twice in one sweep.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review follow-up: detection preferred GOMEMLIMIT outright, so a
GOMEMLIMIT set looser than a tight cgroup limit would size workers past
what the container can hold. All sources — GOMEMLIMIT, the effective
cgroup limit, host memory — are now consulted and the smallest positive
one wins, with the min logic extracted and unit-tested alongside new
sub-1GiB sizing cases. The floor of 2 workers is kept deliberately: a
sub-1GiB deployment already ran 2 workers before this branch, so the
floor preserves the shipped baseline rather than regressing below it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…scode-crackle

fix(playback): stop Firefox crackle after AAC transcode

- A reorder dropped on the grid's trailing area appends instead of
  silently doing nothing, with the same "after the last widget"
  indicator the sheet-add path shows; mouse users could otherwise never
  move a widget into the last position.
- The playback-activity buckets and reliability scalars read one
  repeatable-read snapshot, so a session starting or finalizing between
  the two statements cannot make them describe different session sets —
  and now(), the transaction timestamp, gives the bucket filter and the
  reported window the same instant.
- The reconciler and cleanup publish the playback cache invalidation
  whenever sessions changed, not only when the events hub is absent:
  the hub feeds connected admin clients while the bus invalidates the
  playback-derived admin aggregates across nodes, so gating one behind
  the other left those caches TTL-only in the normal configuration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

imageutil.Thumbhash decoded the full original bytes in pure Go before
scaling to 100px, materializing the whole raster on the heap — a large
provider poster reaches well over a hundred MiB — once per concurrent
image-cache worker. The memory-efficient path already existed:
normalizeThumbhashSource shrinks to 100px through libvips with
shrink-on-load, but only ran as a fallback when the Go decode failed.

The order is now reversed: vips downscales first and Go decodes the
tiny PNG, with the pure-Go decode of the raw bytes kept as the fallback
for anything vips cannot parse. Hashing a 6000x4000 JPEG now allocates
about 0.5 MiB on the Go heap instead of over 36 MiB, pinned by a test.

The emitted hash bytes change for a given image. Stored thumbhashes
remain valid placeholders, and the one equality-comparison site — ebook
scan cover change detection (internal/scanner/ebook_scan.go) — stores
the freshly computed hash whenever it re-caches, so the change costs
one re-cache per scan-covered ebook on its next scan and then
converges; it cannot loop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- The leaderboards' watched-time CTEs window on ended_at — the same
  stop instant watched_at records — so a session straddling the window
  boundary counts toward both plays and watch time.
- The sampler sweeps the telemetry registry instead of snapshotting it:
  a snapshot reports totals as of the last telemetry sweep, and a sweep
  interval above one minute would read zero growth between sweeps and
  then attribute several minutes of bytes to one, spiking the chart.
- Each sampler tick's database work runs under a 30-second deadline, so
  a wedged pool costs missed minutes instead of silently stopping
  sampling and retention pruning for the length of the outage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Inserting media_items rows with local cached poster paths fires the
reopen_image_ladder_backfill_v2 trigger; on a database that has
completed ladder v2 that lowers the image_ladder_backfill_state
singleton, and deleting the fixture rows does not restore it. Both
tests that seed such rows now snapshot the singleton and restore it
last (t.Cleanup runs LIFO, and the poster-row restores themselves
re-fire the trigger).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review follow-up: below 2x the per-worker budget the floor of 2 wins,
which is deliberate — a sub-1GiB deployment ran 2 workers before this
sizing existed, so the memory cap never reduces a host below its
long-standing baseline. Say so on the function instead of leaving the
budget to read as a guarantee.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…trigger

reopen_image_ladder_backfill_v2 declares a plpgsql variable image_type
while its manifest probe filters on
artwork_revision_gc_candidates.image_type; the bare right-hand
reference in "manifest.image_type = image_type" matches both, and
plpgsql's default variable_conflict=error raises 42702 the first time
the probe runs. The probe only runs once backfilled_version reaches 2,
so on a v2-complete deployment every local cached-path publication on
the trigger's tables fails outright instead of reopening the fence —
the exact writes the fence exists to catch. Surfaced by
TestImageLadderBackfillLateOldArtworkReopensCompletedVersion, which
fails on main against a migrated database (CI never runs DB-gated
tests).

The new migration re-creates the function with the variable renamed;
behavior is otherwise identical, and the Down restores the previous
definition verbatim.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…cache-concurrency

perf(metadata): scale image-cache concurrency with host CPUs

Displacing a cached poster path ending in /original.<ext> fires
queue_displaced_artwork_revision, which inserts an
artwork_revision_gc_candidates row or resets an existing candidate's
schedule, attempts, lease, and error state. The media_items row restore
alone does not undo that. Both poster tests now snapshot the candidates
for every displaceable path before running, delete candidates the reset
or the fixtures created, and restore pre-existing candidates
column-for-column. Verified with decoys: a candidate with distinctive
attempt/lease/error state survives a test run byte-for-byte, and a
displaced row that had no candidate ends with none.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…k-reconcile-resets

perf(metadata): batch artwork reconcile bulk resets

…ladder-ambiguous-image-type

fix(metadata): rename ambiguous image_type variable in ladder reopen trigger

…dmin-dashboard

feat(admin): customizable widget dashboard with insights, metrics sampler, and server-side layouts

…wnscale-before-decode

perf(images): downscale via libvips before decoding for thumbhash

- Display the ordered build number alongside the version
- Fall back to the version when no build number is available

…-number

feat(admin): show build number in health strip